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ABSTRACT 


This study evaluates the usability of a U.S. Navy 
Decision Support System (DSS). The DSS was developed to 
enhance the performance of tactical decision-makers 
within a Navy Combat Information Center. The goals of 
this study were to test the DSS against usability 
criteria and objectives to track future redesign efforts 


and system improvements. The purpose of this analysis 


was to (1) assess the system's usability, (2) identify 
problems areas in the graphical user interface, (3) 
report trends in user feedback, and (4) provide 
recommendations addressing major usability issues 


encountered by participants. The study tested whether 
the DSS met the usability objectives of (a) 90% 
successful task completion, (b) ease-of-use ratings of 
somewhat easy or better, and (c) satisfaction ratings of 
somewhat satisfied or better. The DSS did not meet these 
usability objectives for task completion or ease-or-use, 
however the DSS did meet the usability objective for 
user satisfaction. All participants reported that they 
enjoyed working with the DSS and believed that it would 
be a significant step forward in information management. 
Based on the usability data gathered in the study, 
recommendations are provided to address the usability 


Tceues. 
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EXECUTIVE SUMMARY 


This study evaluates the usability of a U.S. Navy 
Decision Support System (DSS). The DSS was developed to 
enhance the performance of tactical decision-makers 
within a Navy Combat Information Center. The DSS is 
still in the development phase and has continually been 
improved based on empirical studies and subject matter 
expertise. The most recent prototype version, known as 
the DSS-2, is the focus of this study. The goals of this 
study were to test the DSS-2 against usability criteria 
and objectives to track future redesign efforts and 


system improvements. The purpose of this analysis was to 


(1) assess the system's usability, (2) identify problems 
areas in the graphical user interface, (3) report trends 
in user feedback, and (4) provide recommendations 
addressing major usability issues encountered by 


participants. The study tested whether the DSS met the 


usability objectives of (a) 90% successful task 
completion, (b) ease-of-use ratings of somewhat easy or 
better, and cc} Satisfaction ratings of somewhat 


Satisfied or better. The DSS-2 did not meet these 
usability objectives for task completion or ease-or-uSe, 


however the DSS-2 did meet the usability objective for 


user satisfaction. All participants reported that they 
enjoyed working with the DSS-2 and believed that it 
would be a significant stepeetenr@erd in) CIC ameermacien 
Management. Based on the usability data gathered in the 
Study, recommendations are provided to address’ the 
usability issues. 

The methodology applied in this study was useful in 
the evaluation of the DSS-2. This study demonstrated 
that traditional human-computer interface usability 
methods could be directly applied the evaluation of 
synthetic environments. The DSS-2 is a simple synthetic 
environment represented on two computer monitors. Given 
the success of this methodology with the DSS-2, it would 
be appropriate to use this methodology in evaluating 


more complex synthetic environments. 
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rT. INTRODUCTION 


A. OVERVIEW 


The United States Navy and Marine Corps strategy of 
forward presence suggests that they will be first on the 
scene in times of crisis. Furthermore, since a majority 
of the world's population lives within 200 miles of the 
ocean, most future contingencies are likely to involve 
littoral warfare (Mundy, 1994). This strategy will 
increasingly place Naval forces in coastal areas where 
they will be forced to operate in confined and congested 
areas (Hueery rns , Kelly, & Morrison, 1997). These 
constraints will result in operational scenarios that 
will require both increased information processing and 
accelerated decision-making. These challenges are 
exacerbated by the fact that current real-time battle 
management systems are primarily effective in dealing 
with all-out conflicts and not particularly capable in 
Situations, such as littoral conflicts, where human 
intervention in decision-making is more GaG ieee aa 
(Hutchins, Morrison, and Kelley, 1996). 

Two unfortunate and well-known incidents involving 
EMeGmUvo,S. Seark ana the U.S:S. Vincennes highlight this 


challenge. The U.S.S Stark incident centers on a 


decision made by the commander not to engage an inbound 
aircraft. The aircraft was not considered a threat by 
the commander, however it was a threat. The aircraft 
Significantly damaged the U.S.S. Stark and numerous 
lives on board were lost. The commander of the U.S.S. 
Vincennes faced a similar problem, yet believed the 
inbound aircraft he faced was a threat to his ship. As a 
result, the inbound aircraft was destroyed by the U.S.S. 
Vincennes. The aircraft turned out to be a commercial 
passenger airline and all lives onboard the airliner 
were lost. The Tactical Decision-Making Under Stress 
(TADMUS) program was initiated to address these types of 
problems. The principle product of the TADMUS program is 
the Decision Support System (DSS). Due Ee the 
criticality of these issues and the need to correctly 
identify threats, it is imperative that the design of 
the TADMUS DSS system be intuitive and easy to use. This 
study will evaluate the usability of the TADMUS DSS 


graphical user interface. 


B. BACKGROUND 


The DSS system was developed to enhance the 
performance of tactical decision-makers. It was derived 


from current cognitive theory. This derivation first 


analyzed the cognitive tasks performed by decision 
makers in a shipboard Combat Information Center and 
second, developed a set of displays to Support these 
tasks based on the underlying decision making processes 
(Morrison et. al., 1997). The DSS 1S currently a 
prototype and is planned to be formally tested onboard a 


Navy ship in 1999, see Figure 1. 
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Figure 1: TADMUS DSS Integrated Display 


The DSS is still in the development phase and has 
continually been improved based on empirical studies and 
subject matter expertise. The most recent prototype 
version, known as the DSS-2, is the focus of this study. 
A usability test was conducted on the DSS system to 
evaluate human performance and user preferences. This 
test also identifies usability issues that focus on 


future design and redesign efforts. 


C. PROBLEM STATEMENT 


Given the critical nature of the tasks supported by 
the DSS system, “and the impltveations of a Giri reule to 
use design, a usability study was conducted on the DSS. 
The goals of this study were: 

e Conduct a usability study to test where the DSS-2 
product 1s today ory regards to usability 
Ciguteria. 

* traces usability measures (e.g., successful 
completion of tasks, error rate, time to complete 
task, ease-of-use, and user satisfaction) in 
order to track product improvement. 

e Identify usability issues to address future 
design iterations. 

e Provide user feedback to DSS development team. 

e Provide recommendations to address usability 


issues encountered by users during testing. 


D. OBJECTIVE 


The purpose of this study 1s to assess the 
usability of the human-computer interface of the TADMUS 
DSS. The long-term objective is to provide a methodology 
and baseline information for the evaluation of future 


systems. 


E. SCOPE AND LIMITATIONS 


To marrow the scope of the thesis, only human- 
computer interaction performance and preference will be 
analyzed. Issues Gemc cerning conceptual cognitive 


decision-making will not be addressed. 





Ir. LITERATURE REVIEW 


A. OVERVIEW 


The review of literature for this research included 
journals and textbooks covering the subjects et 
usability evaluation, human-computer interaction, and 
synthetic environments. The purpose of this literature 
review is to provide an overview of the historic and 
current theories and practices relating to usability 
evaluation and to provide information on the methods 


used in this study to evaluate the DSS. 


B. BACKGROUND 


There has been significant growth in the fields of 
synthetic environments and usability engineering. The 
term synthetic environment is used to refer to virtual 
reality, virtual environments, teleoperator system, 
telerobotic systems, augmented reality and synthetic 
environments in general. However, these two fields have 
experienced growth independently. WD underlying 
assumption among De a (synthetic environments) 
researchers and developers sometimes seems to be that 
(synthetic environments), because they are a novel and 


impressive technology, are inherently good and usable 


(Gabbard and Hix, 1997, p.3)." Phe tools developed ane 
the lessons learned in the field eis usability 
engineering have yet to be significantly applied to 
synthetic environments and those that have been applied 
typically have not addressed the broad issues of 
usability throughout the system (National Research 
COuneale 1997; and Gabbard and Hixs 1997). The 
integration of these two fields will mutually benefit 
both. Usability engineering will gain a technologically 
Savvy customer and developers of synthetic environments 
will drastically improve the usability of their 
technologically complex, and frequently difficult to use 
systems. 

Usability engineers will need to modify existing 
methods and tools as well as develop new ones 
specifically for synthetic environments. For example, 
typical human-computer interaction usability studies 
focus on standard graphical user interfaces where there 
1S a esingle user. In the synthetic environment, 
innovative and non-Standard methods of interaction as 
well as a multi-user capability call for a redefinition 
of the current usability paradigm. To facilitate this 


transition, Gabbard “ana Hix (S97 =eneve outlamea che 


FOuUr Serimarvyaeeusabaality chamaecteristies related EO 
synthetic environments. These are: 
1.Users and User Tasks in Synthetic Environments - 
general user and task characteristics and types 
of tasks in synthetic environments. 
2.The Virtual Model - usability characteristics of 


generic components typically found in synthetic 


environments. 
32 SyYe@eneeuce Environment User Interface iinyenehe 
Mechanisms ~ usability characteristics Orr 


synthetic environment input devices. 

4. Synthetic Environment User Interface Presentation 
Components - usability characteristics of 
synthetic environment output devices. 

Gabbard and Hix have developed a comprehensive 
taxonomy based on these four areaS in order to move 
beyond the "let's build it and see what happens" method 
that is often employed in synthetic environments. This 
taxonomy is a elassitication, enumeration, and 
discussion of usability issues in synthetic environments 
and was developed to ensure that usability will be 
integrated ee the development of synthetic 
environments. LRcaAcGd tone OumeEhnilS © work additional 


analysis of usability engineering and its integration 


= 


intomesynthetie envastonments throtigheeusabamaeee testimg 
needs to be undertaken. The first step in determining 
how this can be accomplished is to examine usability, 


usability engineering, and synthetic environments. 


Cc: USABILITY 


Usability engineering is a systematic approach to 
usability. In general, usability means that the people 
who use the product are able to do so quickly and easily 
to accomplish their own tasks (Dumas and Redish, 1994). 
This definition is based on four essential points, they 
are: 

1. Focus on users. 

2. People utilize products to be productive. 

3. People have limited time to accomplish tasks. 

4.Users decide when a product is easy to use. 
Usability 1S concerned with the sum total of a product. 
Usability should not only be considered an issue for the 
primary system functionality, but should also be applied 
Ee training materials, help packages, and other 
associated features of the system. In order to improve 
the ease-of-use of a product, usability should be 
considered throughout the development of a system, from 


initial design through final deployment of the system. 
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Dumas and Redish (1994) provide seven principles 

for ens@eingeuseabi lity: 

1. Engineering it into a product through an iterative 
design and development process. 

2. Involving users throughout the process. 

3. Allowing usability and users' needs to drive design 
decisions. 

4.Working in teams that include skilled usability 
specialists, interface designers, and technical 
COMMUNIGAEOTS . 

5. Setting quantitative usability goals early in the 
process. 

6. Testing products for usability, but also integrating 
usability testing with other methods for ensuring 
usability. 

7. Being committed to making technology work for 
people. 

This antegratiwen of usability into a product is commonly 
called usability engineering, (Good, 1988; Whiteside, 
Bennett, and Holtzblatt, 1987). Similar to software 
engineering, usability engineering includes identifying 
users, analyzing tasks, setting specifications, 
developing and testing prototypes, and the iterative 


cycles of development and testing (Dumas and Redish, 


eh 


1994). Gould and ebewis (1935) high teht shoumepeimacwp les 
to facilitate designing usability into products. 

1. Focus early and continuously on users. 

2. Integrate consideration of all aspects of 

usability. 

3.Test versions with users early and continuously. 

4.Iterate the design. 
Identifying usability requirements prior to design can 
save time and money for the deSigner as well as increase 
the likelihood of user satisfaction with the product. 
Systems are developed to help individuals accomplish a 
task. In order to provide a usable system, what the 
individual needs and how they are to accomplish this 
must be ascertained. The primary requirement is to 
understand the prospective users and the audience for a 
system. Dumas and Redish (1994) have identified 
techniques that can be used in a usability engineering 
process. These techniques highlight the importance of 
describing what a person does in their job in terms of 
tasks. When the tasks are analyzed, how the person does 
the  j06, can do the™ 70be ocr Sheuld ado “ime 3505 are 
described (Drury, Paramore, Van Cott, Grey, and Corlett, 


iS 7c 
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Table 1: Usability Engineering Process Techniques 


Techniques for Planning, Implementing, and Evaluating Usability 


Uncovering usability needs before you design 
Identifying users’ jobs and tasks 
Convening focus groups 
Interviewing and observing users in context 
Conducting usability tests of existing versions 
Conducting usability tests of competitors’ products 
Setting quantitative usability goals 

Basing designs on expertise in human-computer interaction (HCI) 
Understanding the HCI and document design approach 
Using HCI and document design principles and guidelines 
Setting and using local rules 

Evaluating usability throughout design and development 
Getting experts to review the design 
Having peers or experts walk through the design 
Having users work with static prototypes 
Having users work with interactive prototypes 
Getting user edits on early versions of documentation 
Conducting iterative usability tests 
Asking users about their satisfaction 

Redish and Dumas (1994) 


In addition to understanding the principles of 
usability and the usability engineering process, it is 
important to set quantifiable usability goals early in 
the design process. By setting quantifiable goals, a 
product development team will have a concrete way to 
measure uSability success. A series of quantitative 
goals with related objectives should be identified prior 
to system development and will facilitate subsequent 
analysis. A team may have a goal to design a product to 
be easy to learn and operate, however this is not a 
quantitative goal and would be difficult to measure. The 
design team needs to define quantitative goals to more 
eaSily measure usability. Subjective criteria can also 


be defined to help evaluate a products’ usability. 


is 


Subjective criteria are often easily derived, yet it can 
be difficult to determine if the criteria have been met. 
Typically, it 1s easier to determine when objective 
goals have been met. However, they may originally be 
more difficult than the subjective criteria to develop. 
For instance, in an air traffic-control synthetic 
environment, a usability goal might be that users should 
be able to detect and identify a new track in less than 
5 seconds. Measures such as these are important in that 
they provide a basis for evaluation on whether the goal 
was achieved, they allow systems to be compared, and 
provide baseline information against which revisions can 


be evaluated. 


D. USER INTERFACE DESIGN 


The usability of a product is inherently tied to 
the user interface. If the user interface is intuitive, 
easy to learn and use, a product will have favorable 
usability ratings. Guidelines and user interface 
heuristics have been established by academia and 
industry experts to best design user interfaces for 
usability. Shneiderman (1997) proposes eight golden 
rules of interface design to best maximize the usability 


of an interface. These include: 
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1. strive for consistency 

2. enable frequent users to use shortcuts 

3. offer informative feedback 

4. design dialogs to yield closure 

5S. offer error prevention and simple error handling 
6. permit easy reversal of actions 

J eeSUppCs = ein ternal Locus GF Contre] 


8. reduce short-term memory load 


ie Sime eeae these rules should foster a sense of 
comprehension and competence among users. This is 
particularly important because users prefer systems with 
which they feel familiar and competent. Furthermore, if 
a user has positive feelings toward a system they are 
more likely to highly rate the performance of these 
systems. These rules were originally developed primarily 
for the standard graphical user interface. However, 
these general underlying principles of interface design 
can be interpreted, refined, and extended to synthetic 
environments. 

Striving for consistency can be problematic in that 
consistency can relate to many aspects of the system 
ae. temumumologa, Golerme, layout, input and display 


formats). For example, consistency in a virtual 
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walkthrough of a house could refer to consistency of the 
visual representation of objects in the environment or 
could refer to consistency of human interaction with 
these features. It is not always possible to maintain 
consistency across all dimensions of a system, but 
identical symbology and methods of interaction should be 
employed throughout. 

Shortcuts enable frequent users to reduce the 
number of interactions required to obtain a desired 
result and also increase the pace of interaction. In the 
synthetic environment, unique methods of input and 
display need to be improved to take better advantage of 
shortcuts. For instance, there may be times when a 
three-dimensional virtual environment could be switched 
to a two-dimensional map on which the user could 
navigate. When the user reaches a desired location, a 
return to the three-dimensional world eeu la be 
initiated. Other shortcuts could include gestures or a 
series of gestures, which are not directly relevant in 


the current environment, but offer a shortciemeno anotnter 


environment. 
Otter tiag intormat ven feedback facilitates the 
user's immersion AP fall synthetic environments. This 


feedback can vary in degree with infrequent and minor 
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ACTIONSMaBeCSUMEING w@n small changesewan ethes synthetic 
environment, whereas critical and major actions result 
in substantial changes in the visual presentation. 
Without substantial information feedback users may not 
be able to fully complete actions or understand their 
current status. For example, in an air-traffic control 
system the selection of an aircraft should be indicated 
through a state change (1.e., the display alters and an 
object becomes highlighted). 

Usability of a system can be further maximized by 
designing dialogs to yield closure. This can be achieved 
by grouping a set of actions to provide a natural flow 
through a users’ tasks. This sequencing of actions 
provides the user better awareness of the actions taken 
and gives the user a sense of closure of the sequence. 
An example of this concept is virtual kitchens where a 
user can pick up a dish, manipulate the dish, and break 
the dish. If correctly employed, the user will clearly 
know the status of the dish and the associated action. 
At the conclusion of the sequence of actions, the user 
will clearly see the dish replaced on a counter or 
broken into several pieces and be rewarded through this 


sense of closure and awareness. 
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Whenever practical, users should be permitted to 
reverse actions if they choose. Users tend to make 
mistakes, therefore a system should be designed to allow 
users to recover from errors easily. This may reduce any 
stress or anxiety the user has when operating within a 
synthetic environment. In the virtual kitchen example 
mentioned above, a user may have selected the "wrong" 
dish. The user should be able to recover from this error 
and replace the dish in the same place as before. 
However, the virtual kitchen example also suggests when 
this would potentially not be allowed. For example, if 
the user has broken a dish, there is no recreation of 
Ene wedash . “Sanawikails; 7 trrin Saimissave fire=econtrol 
station a missile is accidentally fired there is no 
aballii tye ter cal itwhome. 

The design of a system should also support a users’ 
“internal locus of control.” Users should be the 
inswtpwators “of actions Get “ehe respomders Stomractions 
(Gaines, HeS Sul.) When appropriate, a synthetic 
environment should be designed with the users in 
command. Whereas the status of objects Te the 
environment would be appropriately updated and 
maintained without user action, the autonomous movement 


of the user within the environment or a drastic altering 
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om the Visual oricntationmweuld be inappropriate. The 
issue of system interruptions arises in this context 
(McFarlane, 1998). If a user is engaged in the synthetic 
environment, when 1s it appropriate for the user to be 
interrupted with a competing task? How should that 
ipehe-hanaeyohommeyel manifest itself? This issue of 
imberruptions needs further study, in» *spartrecular within 
the context of virtual environments. 

The reduction of short-term memory load is 
essential for optimal integration of a user within a 
synthetic environment. Humans are limited in their 
ability to maintain excessive amounts of information in 
their short-term memory. Designs ite synthetic 
environments should include cues, mnemonics, and 
standardized sequences of actions. Whenever possible, 
access to integrated assistance information should be 
provided. For instance, ILD Sie lia Gat ft he conten o |. 
environment, the history of the air tracks should be 
made available to the user. If a task requires a series 
of actions, a list of those actions should be available. 
When designing systems to reduce short-term memory load, 
the designer should remember that humans have been shown 


to be able to remember seven items plus or minus two. 
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These rules of interface design, which are based 
upon existing usability models, can be modified for the 
synthetic environment. However, the limitations of 
existing usability models should be understood. 
Synthetic environments have many unique characteristics 
and understanding these is essential in addressing the 
overall usability. Unique characteristics of synthetic 
environments not supported in existing usability models 
include perceived presence and perceived real world 
fidelity and existing models do One support 
quantification or qualification of a user’s perception 
of such characteristics (Gabbard and Hix, WOO 7). 
Traditional usability models are also limited in scope 
im that -etheyotypically s@oectise@on assaingle@iwser ata 
Single site. Other issues involve the multiple and 
unique methods of interaction and display that are 


continually being developed for synthetic environments. 


E. USABILITY TESTING 


A usability test primarily measures ease-of-use. 
According to Dumas and Redish (1994), "usability testing 
1s a systematic way of observing actual users trying out 
a product and collecting intcremation about the spec imice 


ways in which the product 1s Gasy on Gittirteude to. jebem 
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Eomuse Sp eze." Dumas and Redish (1994) also identify 
three basic tenets of usability testing. The first is 
that usability testing should be used to diagnose 
problems and not to determine that the product is 
flawless. The second is that usability testing should be 
employed early in the development of a product and 
often. Lastly, that usability testing is part of a 
process that focuses on usability throughout design and 
development. 

In order to best incorporate usability into the 
development process, a thorough testing plan needs to be 
developed. There are several determinants that need to 
be addressed in developing an evaluation plan 
(Shneiderman, 1997; Nielsen, 1993; Hix and Hartson, 
1993; Preece et al., 1994; Newman and Lamming, 1995). A 
foundational determinant is the current stage of the 
design. The requirements for testing an early design as 
compared to a late design will differ in that general 
concepts of user interaction with the design need to be 
tested early, whereas testing of a late design may be 
targeted more at identifying consistency within the 
environment and task completion. Thea i 6m, the 
erteircality of the environment is a Signi oe reant 


determinant in deciding the objectives of the test. The 


Pat 


level of task completion rates and number of errors 
allowed in a test will vary depending on whether an 
environment is being developed as part of a life- 
critical system or as an entertainment system. Finally, 
factors such as the novelty of the project, the number 
of expected users, the time available, the costs of the 
product, the available resources (1.e., time and money 
available for testing), and the experience of the 
usability testers themselves play a role in shaping the 
usability test. 

Usability testing of a system has become essential 
not only to maximize the usability of the system, but 
also to verify that contractual requirements have been 
met and to document that testing has been conducted in 
case legal issues or lawsuitS arise concerning the 
operating of the system (Shneiderman, 1997). This is 
best illustrated when one considers that perfection is 
not possible in any system, particularly systems that 
incorporate human users. The varying degrees for which 
errors will be tolerated relates directly to the 
requirements to bring the system to full operational use 
and the impact that the errors may have aur Limg 
operational use. However as Shneiderman (1997) suggests, 


systems which require high levels of input such as 


ee 


nuclear-reactor-control Or al imatrat fic=eont red: 
emergencies are very difficult to test. However, testing 
methods to deal with stressful situations, which include 
life-critical applications, are increasingly needed. 
Another usability testing method employed EO 
improve a product’s usability 1s an expert evaluation of 
the system. Nielsen and Mack (1994) argue that formal 
expert reviews can generally provide more useful 
information as compared to informal demonstrations to 
colleagues or customers. This requires that expert 
reviewers are available to the usability testing team. 
eis available, expert reviewers can be employed 
throughout the design and testing of a system. The 
typical product of an expert review iS a report 
outlining identified problems and recommendations for 
improvement. The forms these reviews may take include 
heuristic evaluation, guideline review, consistency 
inspection, cognitive-walkthrough, and formal usability 
inspection. Expert-reviews do face challenges. er 
instance, expert-reviewers may be confronted with new 
systems and technology they are not completely familiar 
with and for which they may not fully understand the 


design rationale or development history. However, expert 
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reviews typically provide a fresh look at a system and 


are useful in evaluating system development. 


Table 2: Expert Review Methodology 


Expert-Review Description 
Method 

Heuristic Expert-reviewers critique an interface to 

evaluation determine conformance with a short list of 
design heuristics such as the eight golden 
rules. 

Guidelines The interface is checked for conformance 

review with the organizational or other guidelines 
document 

Consistency Experts verify consistency across a family 

inspection of interfaces, checking for consistency of 
terminology, color, layout, input and 
output formats, within the interfaces as 
well as in the training materials and 
online help 

Cognitive- Experts simulate users walking through the 

walkthrough interface to carry out typical tasks. 
Simulating the day in the life of the user 
should be part of the evaluation. 

Formal Experts hold courtroom-style meeting, with 

usability a moderator to judge, to present the 

inspection interface and to discuss its merits and 


weakness. 


Shneiderman, 1997 


Formal usability testing for laboratories can 
provide information concerning user needs and abilities 
that an expert-review may miss. Usability testing and 
usability laboratories have been developed to capture 
the user experience directly. The information gained is 
used to confirm progress in the design of a system and 


to obtain recommendations to improve upon the system. 


Typaeallyheka Loumaimusabideityeiscibvey.  1seaconaiic heen ea 
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usability TPaboratory mm a centrolle@esetting® with a set 


of tasks for the user to undertake. 


Usability studies do take other forms, such as 
Nielsen’s (1993) discount usability engineering which 
are “quick and dirty” approaches to task analysis, 
prototyping, and testing (Shneiderman, 1997). Field 


studies are another type of usability study which are 
conducted in actual work environments in order to 
achieve realistic, user evaluation. A different approach 
to these traditional methods is to challenge actual 
users of the system to try to break the system, commonly 
called beta testing. By offering rewards to individuals 
who find flaws ina system, developers can speed up the 
Gevelopment process and correct errors that may have 
been missed through conventional testing. Two serious 
flaws with usability testing in general are that it 
emphasizes first-time usage and lacks a comprehensive 
evaluation of the system due to time constraints 
(Shneiderman, WSIS These flaws necessitate that 
usability testing be supplemented with other methods of 


evaluation such as expert-review. 
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F. PLANNING FOR USABILITY TESTING 


When planning a usability test, often the most 
important question, besides what is required, is how 
long the test should take. If the usability testing is 
an integrated part of the design process and is not 
Simply being conducted on a completed system, then the 
test needs to be as short as possible to obtain the 
necessary information - and short enough so that the 
test is not burdensome. This will facilitate the 
iterative nature of proper usability testing. Testing 
length depends on many factors, including how much prior 
testing has taken place, how complex the system is, and 
Penic scope of the system to be tested. Dumas and Redish 
G99 4) suggest @ehat “tradt@ticonal testing lengens@ ral! 
into one of four categories. Organizations that follow 
formal testing and generate comprehensive test reports 
allow eight to twelve weeks. Shortened testing periods 
of four to six weeks are frequently used when there 
exists a strong collaboration between team members and a 
shortened formal report is used. When only a particular 
aspect of a system is to be studied with well- 
established procedures, one week can suffice. Just-in- 
time “testing is “discouraged, but can Stil) "provide 


useful information in a couple of days if necessary. 
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Studies conducted by experienced and dedicated 
individuals who take the necessary time will most often 
achieve the best results. Furthermore, proper planning 
Eiedilco them@Gderanittien of goals and concerns, deciding 
who should participate and recruiting these individuals, 
developing and organizing tasks and task scenarios, 
deciding on usability measures, preparing the test 
Materials and test environment, and conducting a pilot 
test. 

Dumas and Reddish (1994) suggest that defining 
goals and concerns for usability testing can be viewed 
as a three-stage process. The first stage is making 
choices among goals and concerns. For instance, 

Is your main concern whether new users 

will be able to get up and running to 

do basic tasks quickly, oor whether 

users who have had the product for 6 

months can figure out more advanced 

functions? You may be concerned about 

both, but you’ll have to plan two 

different tests to learn about both 

(Dumas and Redish, 1994, p. 111). 
The second stage is moving from general concerns to 
specific ones. This helps determine the type of subjects 
necessary and begins to shape concerns into quantitative 
objectives. Lastly, understanding the sources of these 


goals and concerns allows the usability engineer to 


better develop the testing scenarios and tasks. Some 
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sources include expert-reviews, user feedback, and 
previous tests. 

The decision on who should participate in the study 
should be based on developed user profiles. Ideally, 
user profiles should have been developed prior to design 
of the system and usability testing. If this has not 
been done, a user profile can be developed by 
1dentifying all the relevant characteristics that an 
individual using the system should have. The two primary 
eanienetineniieetin of concern are those that all the 
individuals have in common and those that may make a 
Gifference between the individuals. For example, if a 
command and control synthetic environment were being 
deployed onboard a US Navy ship for the first time, 
certain questions need to be addressed, these include: 

1.Will many users be working with abstract or 
Simulated environments for the first time? 

Paani tt many individuals be experienced with 
personal computer applications, but new to the 
Synthetic environment? 

3.Will many users already be adept at using the 
input devices? 

4.Who will be using this system -- commanding 


officers, junior officers, or enlisted personnel? 
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The decision on how many subjects are required for 
a study can also be a challenging question. Usability 
engineers are often delighted to have ten to twelve 
subjects participate in a study, whereas a statistician 
might insist on no less than thirty-six to forty-eight. 
The realities of time and budget constraints often 
result in usability studies having six to eight 
subjects. A simple answer to the question of how many 
subjects to use is enough participants to complete the 
study as efficiently as possible. 

A method of determining subject size proposed by 
Bailey (1997) is based upon a variation of the binomial 
proba bialmstaeesiio rmudla, 

1-(1-p)* 
where, 


Pp = probability of the event occurring 


n number of test subjects 

For this method, a subject matter expert or team of 
experts must first derive the likelihood, an estimate 
for p, that an element of the system will confuse any 
one test subject. For example, suppose that it has been 


determined by a panel of experts that the likelihood of 


any one test subject having difficulty identifying a 
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confusing air-track® eon mm awsynthetic enwamenmment wis 
0.5, that is to say using ”"a@ single subj eck “in Veur study 
there is a 50/50 chance of the subject having a problem. 
If two subjects are tested, the probability that the 
confusing air-track icon be identified rises to 0.75. 
This »probability ismecabeulated weeing «theses banemaal 
formula presented. Furthermore, if three subjects were 
used, this probability rises to 0.87 and with seven 
subjects to 0.99. Therefore, by basing our calculations 
on — original likelihood, we can determine sample 
size. Table 3 provides a chart for determining sample 
Size. Problem probability can be roughly assumed to be a 
rough estimate of problem severity (Bailey, 1997). 


Table 3: Likelihood of Performance Test Subjects Having 


Problems 

Problem Number of Test Subjects 

Probability a 2 3 4 5 6 7 8 9 10 
OS OO 5eetOae ie. | Soe. Come oe To wore | rae 
5118, Fs een ES ee ey ce ee ee SS 5) 61 36s 
RS Ls LO oe” CC oe Oro oo 8. 3 wid eee 
aD 05 A eS 8 6 Ce 1 ees Oe .90 132 .94 
50 50 S75 “oreo. “Seo7 Ss” 
7S me 94) O86 @ 99 


.90 .90 .99 
Bailey, 1997 


After determining the number of participants and 
the length of the study, the test can be developed. Once 
the goals and concerns of the test have been defined, 


the initial tasks to test should be selected and 
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organized to best address these goals and issues. The 
tasks then need to be placed in a context that is 
understandable EO the user. Tate is through the 
development of task scenarios that this 1s accomplished. 


Mhese scenarmes serve as the basismfior the test. 


G. CONCLUSIONS 


Synthetic environments are an emerging technology 
that will enable individuals to perform new functions 
and accomplish older functions in a new way. Usability 
engineering and usability testing tools have recently 
been developed primarily in conjunction with standard 
graphical user interfaces. If synthetic environments are 
going to mature and become integrated into our society, 
they must be easy to use and enable individuals to 
accomplish tasks more efficiently. This maturation can 
only take place if these fields can be integrated so as 
to establish systematic and standardized methods of 
evaluation. Tees evolution will begin by basing 
synthetic environment designs on human-computer 
PAzeraGCzLon prancipLles. 

PAS thesis argues that the application of 
principles of human-computer interaction derived from 


existing literature and research can be applied to 
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synthetic environments. There is significant enthusiasm 
SUrrOUNndING wsynthetic emvimenments, “Smt wl set Pomect sert 
has been made to mature the field of synthetic 
environments and utilize existing models of usability 
and user interface design. This can be best addressed by 
demonstrating what 1s accomplished when the methods of 
usability are applied to synthetic environments. It will 
only be through a Significant development and successful 
implementation of an existing or emerging synthetic 
environment uSing usability principles throughout its 
design and implementation that this integration will be 


taken seriously. 
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i Es METHODOLOGY 


A. RESEARCH APPROACH 


This study involved the analysis of an existing 
decision support system and the development of 
evaluation methods based on this system. The purpose of 
this analysis was to assess the extent of the system's 
usability, to assess the effect of the interface on the 
user, and to identify any specific problems with the 


Sas temm Daseeet a dij 1997 ).. 


B. DATA COLLECTION 


Participants. 12 participants for this study were 
recruited at the Naval Postgraduate School (NPS) in 
Monterey, California. All participants were military 
officer iMSeruecors Cis students at NPS and had 
previously served as Surface Warfare Officers (SWO). Six 
of the participants had served aboard Aegis ships and 
Six had not. The participants were further divided by 
experience level into one of two categories, low and 
high. Experience levels were based on a combination of 
months spent at sea and the number of deployments. The 
low experience level group on average had 44 months at 


sea and two deployments; the high experience group on 
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average had 58 months at sea and an average of 2.7 
deployments. Participants were distributed across four 
categories. Table 4 shows the distraburwvon of 
participants across these categories. 


Table 4: Subject Distribution 

















Haee i 
experience 


Low 
Experience 





Non-Aegis 
Aegis 


ALT parcieipancts had exper 1enee mas © Omlod ama eiya@ ailchemme id 
Center (CIC) Watch Officers. Five participants had 
additional experience as Tactical Action Officers. Two 
of the subjects were U.S. Navy Lieutenant Commanders and 
ten were Lieutenants. 

Instrument. This study will provide a benchmark 
across usability objectives. A usability task script and 
post-task questionnaire were administered EO all 
Subjects. At the conclusion of the study, a post-test 
questionnaire was administered to the participants. See 
the descriptions below for specific definitions of 
objectives. 


e 90% Successful completion of tasks. 
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e 90% Error free rate. 
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e 90% score of 3 or better on a7 pt. scale (e.g., 
l=easy, 3=somewhat easy, 5=somewhat difficult, 


and 7=difficult) in ease-of-use. 


e 90% score of 3 or better ona7 pt. scale (e.g., 


Il—Satistied, 3=somewhat satisfied, 5=somewhat 
GQlssatistiecd, and f=disscatist teat in user 
Sawaistact1 on. 


Ideally, by the time a Decision Support System is 
released to the fleet, these objectives should be met 
and/or exceeded in order for the system to meet high 
ease-of-use standards. 

Procedure: Participants completed an informed 
consent form and demographic questionnaire (Appendix A 
and B). The participants also received a usability task 
script along with a brief description of the evaluation 
scenario (Appendix C). Participants sat directly in 
front of two 21-inch computer display monitors and 
controlled the DSS-2 with a computer mouse. The 
beginning of the usability evaluation consisted of the 
participants responding to a series of questions 
concerning their initial reaction to the DSS-2 graphical 
user interface (Appendix D). Participants were then 
directed to read aloud and execute the tasks provided 


them in the task script. Following each series of tasks, 
a) 


questions concerning the usability of the DSS-2 were 
presented. Additional questions concerning participant 
satisfaction as well as current understanding of the 
DSS-2 were also presented. Upon completing this phase of 
the study, participants were timed on the completion of 
tasks using the DSS-2. The study concluded with the 
administration Ot a post-test questionnaire. 
Participants received no training on the DSS-2 prior to 
the usability study. The DSS-2 component names, such as 
track profile and response manager, were not used during 
interactions with the participants. These components 
were addressed in respect to the location they would be 
found on the display monitor. For example, the track 
profile component would be referred to as the area in 
the upper left side of the right monitor. 

Throughout each usability session, the following 
measurements were taken during the performance of user 
tasks. These measurements were used to assess whether or 
not each usability objective had been met. These 


measurements include: 


e Task Completion Rate: The proportion of participants 
who complete the task successfully and independently 
without critical errors. A critical error has 


occurred when the participant either requests 
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assistance from the usability engineer or commits an 
uncorrected error that results in an incorrect 
outcome for the task. 

Error Free Rate: The proportion of participants 
completing the task without any errors, critical or 
non-critical. Non-critical errors include any error 
corrected by the test participant without 
intervention by the usability engineer or an error 
left uncorrected, but which does not affect the 


correctness of the outcome of the task. 


User Satisfaction: The User Satisfaction rating is 
derived from a series of questions which the user 
rates on a 7-point' scale, ranging from very 
dissatisfied to very satisfied. The questions 
solicit user opinions with regard to ease-of-use, 
Simplicity of the human-computer interaction, system 
functionality, and general satisfaction with the 


DEeOduce. 


DATA ANALYSIS 


The occurren@en Cte terENOENNEeIe “measurements listed 


above was recorded in a spreadsheet. These data included 


any associated user-feedback information associated with 


the measurement. Frequencies of the various measurements 


a7 


in the database were determined, both in aggregate and 
by measurement type. The categorization of participants 
by experience level and whether they had previously 
served onboard aegis ships was used in presenting the 
results. However, due to small sample size and no 
noticeable differences between categories all subsequent 


analysis was performed on all participants as a single 


Groups 
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IV. USABILITY EVALUATION RESULTS 
A. BACKGROUND 


The results of this usability evaluation are 
presented in the same order they were collected. The 
participant’s initial impressions of the DSS-2 graphical 
user interface are presented along with the participants 
initial impressions of the six major components. The 


DSS-2 components include the Figures 2 through 7: 
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Figure 2: Toolbar 





Figure 3: Geoplot 
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Figure 4: Minicros 
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Figure 6: Response Manager 
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Figure 7: Track Summary 
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The participant’s task completion rates, post-task 
question responses, and the answers to the post-test 


questionnaire are also provide. 


B. INITIAL IMPRESSIONS 


Overall, participant’s first impressions of the DSS 
were positive (Table 5). Participants generally found 
the DSS to be a familiar interface that contained more 
information than they were accustomed to in existing 
shipboard systems. In addition, participants stated that 
their initial impression of the DSS-2 was that it aided 
Situational awareness and 1S used to consolidate 
information. In particular, participants were familiar 
with the geoplot map display and understood that the 
toolbar would be used for manipulating the geoplot map 
(Tables 6 and 7). Participants generally understood that 
the minicros were summaries of ieee clue! track 
information and that this information was ordered in 
some manner (Table 8). Three participants either did not 
know what the minicros would be used for or incorrectly 
identified the meaning of the minicro functionality. 
Panic pantc hada Cdiliireculty identifying the Emeck 
Ppeorrlemcompeonemeror the DSS-2 (Table 9). A majority of 


participants incorrectly believed the Track Profile 
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component to be a weapons status monitor that reflected 
the status and quantities of weapons available. The 
response manager was unfamiliar to all participants 
(Table 10). However, the concept of the response manager 
was generally understood by all Participants. 
Participants stated that the response manager would deal 
with engagement orders, doctrine, rules of engagement, 
recommended actions, os a eneckilast. Generally 
participants correctly identified the track summary 
component, however three Dertwcipeants INGcOrrectly 
assumed that the track summary information pertained to 


own-ship status (Table 11). 
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Table 5: What is your first impression of what you see? 





Familiar Geoplot. 
e Like it, layout. Black and white stand out. 
e Similar to JOTS display. 


e Where we are and where our battlegroup is. Focus on 
Situational awareness, where we are. 


CPi rer eu t,. 

e Situational Awareness. 

e A lot of crap on the screen, cluttered, overwhelming. 
e Lot of information, used to pick out symbols. 

e Consolidated a lot of info. 

e Immediately obvious, right sight. 


e Looks cool. Intimidating. 
Like it, used to one screen. 










Cet More tnre wlen three 





Table 6: What does the far left side of the left monitor 
represent to you? (Toolbar) 


Control panel for display and tracks. 
Power point. 
View of situational picture. 


Track contact info, select track symbology, make 
decisions for you. 


Legend. 


Filter setting. 
Legend for map. 
NTDS notations. 
Toolbar for geoplot. 


Alter geoplot. 
Select what you want to look at. 
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Table 7: What does the upper left area of the left 
monitor represent to you? (Geoplot) 


Map. 

JOTS display, NTDS. 

Where I am and what my radar knows. 
Visual display of geographical picture. 
Operating area, tactical area. 


Geoplot. 

Northern Persian Gulf. 

Geoplot of surface and air contacts. 

Threat access, radar responsiveness in a certain area. 


Big picture, where we are. 
Topographical map. 





Table 8: What does the bottom row on both monitors 
represent to you? (Minicros) 


Quick Summary to what you are seeing. 

Classification of targets, not sure how it classifies. 
What I know of the tracks in my area. 

Contact information, track #’s and names. 

Weapon employment areas. 

Contacts. 


Information of ships. 

Current tracks. Air and merchants set as priority time or 
threat. 

Contact bearing range, speed, and sensor types. 


One for each track. 
Nothing, selection buttons? 
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Table 9: What does the top left area of the right 
monitor represent to you? (Track profile) 


X axis questions and range. 

What our weapons are. 

Status of weapons systems. 

Weapons status, a horizontal bar chart. 
TAO stuff. Order of steps, things to do. 


Nothing, weapons status? 

My weapons and how much I have. 

Our weapons. 

How much I have, status, and range. 


Not quite sure, our weapons status? 
Weapons status. 





Table 10: What does the middle-left area of the right 
monitor represent to you? (Response Manager) 


Engagement orders. 

When things happen, what should happen at COI/ already 
taken place. 

Never seen anything like it. A decision matrix for ROE, 
possible defenses, weapons posture. Where we are and what 
we should do. 

Decision-making. Decisions I need to make. 


I don’t know, maybe doctrine statement. 


Doctrine. 

Distance time line, envelopes. Need to do something. 
ROE s. 

Priority of actions, Recommended actions by system. 


Don tmeaow, a COntinmum Of toedo’ s? 
rine leer. 
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Table 11: What does the top right area of the right 
monitor represent to you? (Track Summary) 


Track details. 

Close control, more information, emitter age. 
@entact COl. 

Has a tracks detailed information. 


Specific track information. Don’t know how it is 
different from (minicros), maybe more specific. 


All I need to know to launch. 


Own ship, don’t like presentation. 


Ship status. 

Contact information. 

Blow up of minitrack status. Active/intel/last known 
DOS?TtL ron. 

Our own ship status. 

Ships information and dependent on current selection. 





Once data was gathered on the participants initial 
impressions of the DSS and its components, a series of 
tasks were presented. Participants completed the tasks 
and answered post-task questions concerning ease-of-use 
and satisfaction. Task completion errors were recorded 
and task completion percentages were calculated. Task 
completion rates which were below the US a mates 
criterion of 90% are highlighted in Table 12 and 


discussed in Chapter five. 
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C. TASK COMPLETION 


Table 12: Task Errors and Completion Rate 


Completion 
Rate 


67% 


TASK Errors 


Task 1: Display the track numbers of all contacts in 
the map display. 

Task 2: Locate and select track number 7012. 

Task 3: Change the map to display the directions all 
tracks are moving. 

Task 4: Remove all unknown tracks from the map 
display. 

Task 5: Display all surface unknown tracks. . 75% 


Task 6: Display all tracks. Pace 1 Gos 


Tack=mchangelehe Esaizemorethemmap Combetter sceeutme 
tracks displayed. 

Task 8: Identify the range of map sizes that are 
available. 


Task 9: Explore these two buttons. When finished 100% 
exploring, select map size of 128 nm. 


Task 10: Please point to own-ship on the map display. 
Task 11: Identify which tracks are potential threats. | 6 | Sa | 
Task 12: Determine whether track 7013 is within its 5 75% 
weapons release range. | 
Task 13: Determine whether track 7013 is within own- 
ship’s weapons envelope. 

Task 14: Identify the bearing and range of track 
mOL6. 


LOO'S 
92% 





100% 

















92% 


100% 


Task 15: Check and see what the most recent warning 75% 
information is regarding track 7011. 
Task 16: Identify the most recent warning information 83% 


fEOpetiaack 7 0 11. 


Task 18: Locate own-ship position on the right 2 83% 
monitor. 
Task 19: Using information available on the right 75% 





Nonttom, —Gderermine, whether track 7Ol6G is within its 
weapons release range. 


100% 


Task 20: Using information available on the right 
monitor, determine whether track 7016 is within own- 
ship’s 5/54 guns weapons envelope. 

Task 21: Select track 7017. 

Task 22: Click on verify airspace, issue a level 1, 
and change CIWS to auto/ready. 

Task 23: Select track 7012. 


100% 
92% 





100% 


Figures 8 through 26 show participant responses to 
each of the post-task questions. Each figure shows 
either ease-of-use or satisfaction ratings for all 12 
particapanves. The bars VT the figures represent 
individual participant ratings. These bars are grouped 
according to the category of user and the bars within 
each grouping are order according to relative experience 
levels within the group. Tables 13 through 43 summarize 
participant comments and what participants thought could 


make the completion of the task easier. 


48 


D. POST-TASK QUESTION SUMMARIES 


on Oo 


pp 


Ratings 
(1-easy, 3-saomewhat easy, 5-somewhat difficult, 7-difficult) 


Nm Go 


ah 





Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 8: How easy/difficult was it to display the track 
numbers? (Question 1) 


Table 13: Comments & what would make this easier? 
(Question 1) 


Madwere Look for it. Training: 
Clutters display, lose visual reference of contacts. 
Just a little more time to study screen. 


Very easy considering ist time use. Change buttons to 
read easier (e.g.) "Display all track #'s". 


Shift click or mousedrag over all contacts. 
Different color background to make it standout add to the 


"show" title for instance "Display on Map". 

I was not sure what I was doling was going to work the 
first time. Experience with the interface. 

Took a step and had to decide if it was a collective 
action or serial. Pre-knowledge. 


Easy once I knew what to do. 
Advanced training, change label to "show all tracks" or 
something like that. Experience. 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difflcult) 





Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 9: How easy/difficult was it to find track number 
7012? (Question 2) 


Table 14: Comments & what would make this easier? 
(Question 2) 


Put track #’s in order on bottom display. 

What will happen to display w/in traffic zones, >20 
COneaces:. 

No problem. 

Need a 10 key to enter track #. 


Experience with the interface. 
Track number hides symbol. If the tracks exceed the 
display. Area, may need a summary list that is 


categorical. 
If I understood the ordering of the track #’s at the 


bot tem. 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 





Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 10: How easy/difficult was it to read the track 


number on the map display? (Question 3) 


Table 15: Comments & what would make this easier? 
(Question 3) 





Possibility of having #'s separate. This has its 
disadvantages too! 

On large scale chart hard to distinguish individual #'s 
Make larger. 

Looking for track # highlighted. Arrange tabs by other 
"SOimES 2 

Flashing contact # or something to make it stand out - gets 
larger for example. 

How far from screen will the user be? More than one user 
may have to use a display. 













Sa 
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i 


Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-dlfficult) 
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Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 11: How easy/difficult was it to display the 
course leaders in the map display? (Question 4) 


Table 16: Comments & what would make this easier? 
(Question 4) 


Used history function first. 
Display all buttons. 


Experience. 
Experience. 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 





Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 12: How easy/difficult was it to remove unknown 
tracks from the map display? (Question 5) 


Table 17: Comments & what would make this easier? 
(Question 5) 


HaCGneGr Glrek Co determine 2f oOn/ort: 
Did it individually by air, sfc, sub, not in one step. 
Little time needed to understand there were toggle 


DUT EONS - 

Very easy since I had the experience of the previous 
task. 

Experience. 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 
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Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 


Participants 


Figure 13: How easy/difficult was it to display all 
surface unknown tracks on the map display? (Question 6) 


Table 18: Comments & what would make this easier? 
(Question 6) 


e Symbology is a plus. 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 


1 





Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 14: How easy/difficult was it to understand when 
the track type buttons were selected? (Question 7) 


Table 19: Comments & what would make this easier? 
(Question 7) 


Training. 
Noise was good indication, but won't work in shipboard 


environment. Distinct color change would be better, (e.g. 
black vs. white). 





SS 





Question 8: How would you explain what this does? 


Table 20: Comments & what would make this easier? 
(Question 8) 


Zooms by tactor Z. 
Goes down to next pre-selected button and increases by 
1/2 the current range. 


Decreases range scale. 

Decreases map size, smaller range scale. 
"+" = Zeom tin. 

ZOOM Imm 

Zoom ie 

Zooms out, lowers scale. 

Zooms in range on increment per click. 
Increase zoom. 


Zooms in range on increment per click. 
Zoom in. 








Question 9. How would you explain what this does? 


Table 21: Comments & what would make this easier? 
(Question 9) 


Unzooms by factor 2. 

Decreases by 2 the current range. 
Increases range scale. 

Increases map size, larger range scale. 
[-)=Zoo0m .OUc- 

Zoom out. 

Zoom out. 

zZooms in. 

Zooms out range on increment per slide. 


Decrease zoom, increase size of area covered. 


Zooms out. 
Z2o0oms out. 
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Question 10. Does the positioning of these buttons 
(plus on the left / minus on the right) meet your 
expectations? Five participants responded yes, seven 
subjects said no. 


Table 22: Comments & what would make this easier? 
(Question 10) 


Numbers decrease to the left, therefore sign should be to 
the left. 


Expect minus on left associate small w/left, large 
WN /Gaallejale 2 

At first I had them reversed. 

"+" on the Right? 


I expected the normal "-" to the left though I saw after 
trial that it means to increase the scale. For me it was 
a compatibility error. 


Would help to have a drag zoom to center and zoom. 
Seems like "+" should be on the right, "-" on the left. 
Prefer to be able to select my own choice of 
Magnification. 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 
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Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 15: How easy/difficult was it to identify own- 
ship on the map display? (Question 11) 


Table 23: Comments & what would make this easier? 
(Question 11) 


Standard NTDS symbology used. 
Hook self. 
Provide legend in Track types for own ship or the 


different nodes show that this is a geo-center display. 
IT assumed that the viewer could always choose the center 
of the screen wherever he wants. 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 








Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 16: How easy/difficult was it to identify whch 
tracks were potential threats? (Question 12) 


Table 24: Comments & what would make this easier? 
(Question 12) 


Standard NTDS Threats are red, unknown white, etc. 

Color on model is yellow/expected NTDS standard of red. 
Either use NTDS colors and symbols or more time and 
familiarity with model. 

Color is deceiving, I was drawn to colored icons, not all 


threats a TGT that had not had any additional evaluation, 
would not be colored, and could be missed. What is a 
threat in this scenario, can I change that criteria and 
then display them? 

This is ambiguous at best. In Gulf all tracks are 
potential threats. 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 
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Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 17: How easy/difficult was it to determine 
whether track 7013 was within its weapons release range? 
(Question 13) 


Table 25: Comments & what would make this easier? 
(Question 13) 


Comparing to left screen made it clearer. 
Multiple envelopes are going to be hard to see. 


This information 1s only based on an assumed 
configuration, may lead to wrong decision making. 


Not sure what the red and gray grids mean. This 1s really 


just a training issue, but easily learned. 

Experience with the interface. 

Pre knowledge of red versus white would make it easier. 
Difficult to Llguré 1 out eene E£1TSst eamewerenamesoutrton 
to say "weapons ranges". 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 
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Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 18: How easy/difficult was it to determine 
whether track 7013 was within own-ship's weapons release 
range? (Question 14) 


Table 26: Comments & what would make this easier? 
(Question 14) 


Comparing to left screen made it clearer. 

Multiple envelopes are going to be hard to see. 

This 1s known info, easy to decide. 

This is really just a training issue but easily learned. 
Because I was guided to the weapons button on the left 


screen. 
Pre knowledge of red versus white would make it easier. 
Difficult to figure it out the first time. Rename button 
to say "weapons ranges". 

Experience. 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 
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Participants 


Figure 19: How easy/difficult was it to identify the 
bearing and range of track 7016? (Question 15) 


Table 27: Comments & what would make this easier? 
(Question 15) 


Familiarity. 

Bottom tab not so clear. Bearing =??, Range=??. 

I need compass display, not for this, but relative 
positions for ship's head for weapons envelopes. 


Not readily recognizable, but it is only due to 
familiarity with the display. 

Titles for course and range displayed. 

Unsure whether 312/16.6 represents range/brg or cse/sro. 
Label the names. 
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Question 16. Do you think there is any meaning to 
the ordering of these items? Eleven participants said 


yes, one said no. 


Table 28: If yes, what do you think the ordering means? 
(Question 16) 









Range and threat. 

e Threat. 

e Threat level. Unsure of ordering within categories 
(Threat, Unk, Nuet) no apparent categorization by 
platform. Appears to list closest contact first. 

e Organized from higher potential threats to lower 
potential threats. 

e Yes, excellent quick reference for track data. Ordering 
acGGemeing to threats: 

e Don't always know what is driving the ordering. 

e Higher threats on left. 











e Threat priority. 
e Potential threat order. 


e I think they are ordered in terms of threat. 
Ordered by threat/unk/friend and then by contact order. 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 
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Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 20: How easy/difficult was it to identify the 
most recent warning information for track 7011? 
(Question 17) 


Table 29: Comments & what would make this easier? 
(Question 17) 


Familiarity and knowing "alerts" had to be clicked on the 
symbol. 

Training. 

Unfamiliar with pull down menu. Not require continued 
user interface with mouse to view alert menu. 

My Understanding of Question? Button to read "Warnings" 
Track updated, I did not see and called wrong 
information!!! 


What are these Mumbers, Tf 16s Lime ian meeds to, Dewin 
00:00:00 format. 


Easily found and understood. 

Except I had to hold the mouse button down to view 
alerts. Menu should stay up after clicked. 

Pre knowledge. 

Didn’t know where to find it. Maybe a flashing warning 
dog: 

Experience. 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 





Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 21: How easy/difficult was it for you to select 
the alert button and view the alert window? (Question 
18) 


Table 30: Comments & what would make this easier? 
(Question 18) 





Familiarity. 

e Unfamiliar with pull down menu. Not require continued 
user interface with mouse to view alert menu. 

e Didn't catch my eye right away. 

e Couple times to learn to hold down. 

e Kind of small. Keep alert up, let any button close 
window. 

e Easily found and understood. 


e Except I had to hold the mouse button down to view 
alerts. Menu should stay up after clicked. 
I would like it to stay in view when clicked. 
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ree tion 19: Click and hold the alert bueecme daa 
do the numbers on the right mean to you? £Seven 
participants believe the number on the eLOne 
corresponded to the time elapsed since warning occurred 
and one the time the warning occurred. 


Table 31: Comments & what would make this easier? 
(Question 19) 


Blue is significant, gray contact updates. 


Lt. Blue means increase threat issue. 

White: general info, Blue: ROE specific info. 

Green appears to be a more serious alert. 

Blue=Warning or threat, Grey=neutral info. 

Blue is associated attack/defending. 

Color means new alert, use red. 

Blue means what affect if has on me. Grey is general 
information. 

Contact within weapons range, I would expect that the 
colors would change (maybe to red) as the contact gets 
closer. 

White general, blue warning, red? Perhaps hostile action. 
Blue means high significance. 

White-narrative, Blue-threat. 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 
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Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 22: How easy/difficult was it to understand what 


the numbers in the alert window mean? (Question 20) 


Table 32: Comments & what would make this easier? 
(Question 20) 






e I just didn't know when I saw them, time since report is 
new to me. 






e Training. 

e Still unsure. Familiarity with system or standard use of 
zulu time on all areas. 

e Column header = Time Elapsed. 

e Both actual and elapsed time on target. 

e 00-00-00. format. 


e Use a plus symbol next to the time to indicate how long 
Since time zero. 
No indication of what they mean. Labeling. 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficuit) 





Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 23: How easy/difficult was it to understand what 
the colors in the alert window mean? (Question 21) 


Table 33: Comments & what would make this easier? 
(Question 21) 


I am still not sure. Familiarity, if I used the system 
the colors would be easy to remember. 


Add a feature for contacts weapons release range in RED. 
Need to find pattern. Use more than 2 color if it is 


going to colorize. 


Use red. 
IT had to think abewtem = 


Yellow is a better warning color, blue is too passive. 
No indication of what they mean. Labeling. 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 
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Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 24: How easy/difficult was it to determine 
whether track 7016 was within its weapons release range 
using the infromation displayed in the right monitor? 
(Question 22) 


Table 34: Comments & what would make this easier? 
(Question 22) 






Familiarity. 

e Training. 

e I need a representation giving weapon's masking areas 
unless I know the graphs account for it. 

e History is less important than current velocity for 


weapons. 
I'm not sure how to interpret the graph. Training. 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 





Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 25: How easy/difficult was it to determine 
whether track 7016 was within own-ship's weapons release 
range uSing the information displayed in the right 
monitor? (Question 23) 


Table 35: Comments & what would make this easier? 
(Question 23) 


Tralining-but by now I have some. 

Have available weapons as a pull down menu. Click 
weapons, release mouse, see choices, make choice by 
clicking desired weapon. 


Select and drag is hard, select/select. 
Basiily understood: 
Make weapons window a pop down and stay vice hold down 


and find. 
I'm not sure how to interpret the graph. Training. 
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Table 36: What do you expect the ASPECT button to do? 
(Question 24) 





Show angle on bow for TGT and TGT angle 

e Show highlighted COI target angle, heading, altitude, etc 

e I don't know 

e 2=D shift by 90 degrees or 180 degrees. 3-D view 
possibly? 

e TGT aspect 

e Target w/relation to ship's head. 

e Change from center on me to a center on him. 

e Place threat contact at (0,0) axis. 

e Weapons release envelope of current velocity of contact. 










e I don't know 
Targetwaspect. 






Question 25: The aspect button only met one 
participants expectations, ten said that it did not. 


Table 37: Comments & what would make this easier? 
(Question 25) 









It tells own ship aspect to TGT, which is easy, and now I 
expected target angle. 


e It's a good tool, but w/"aspect", I expected to see "nose 








On” “Grossing”; eter 
e It give whether your weapons engagement criteria are met 
Or nee. 






e Took me a few seconds to understand what it was telling 
me. Also, the weapons button is hard to use. 

e What's it mean? 

e Needs to be a lot bigger, should be scalable like map, 
should be primary weapons display. 







e No reason, I'm just off. 


ent ote cthamicrthat 1tewoula GO what tt Gla Until © tried 
1t,@ane thenmmt Qiade sense. 







e It iS a weapons engagement aspect for own ships weapons. 





e Because I didn't know what it was for. 
e Does not give me target aspect, provides weapons I can 
use against target with my present heading. 






Fins 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 


NO GO 
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Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 


Participants 


Figure 26: How satisfied/dissatisfied are you with the 
ASPECT button feature? (Question 26) 


Table 38: Comments & what would make this easier? 
(Question 26) 


I like it! 


Satisfied with the information it is giving me, not with 


its ease-of-use. 


Excellent! Rename button to "Arc of Coverage”. 


Bad button name, it 1s 
worry where we can not 


shoot. Change color if 
Once I understood what 
excellent. 

Needs to be in a bigger window and maybe a different 
name. 

Should be bigger. 
Not what I expected. 


showing weapon cut outs, we all 
shoot and how to maneuver to 
inwand*6nc,~.Jedgewds se uitiinatvon . 


it meant, this feature is 





ye2, 


Table 39: What would you expect to do with this area 
based on the track selected? (Question 27) 


Use it for reports and ROE 

Go down the check list to see what's been done 

Time line for deconfliction, verification of COI, weapons 
status upgrade, countermeasures and reports. Basically 
how to classify or fight this target according to ROE. 
Interact/warning 

Line coincides with doctrine packages (I.e. follow these 
doctrine) 

Checklist. 

Conduct doctrine based query factors. 

Perform the actions that the vertical line is in. 

I think that I can click on what warnings I have given 
and the display will change colors to remind me later. 


Respond to recommendations. 
Do the action when the line reaches the left side of the 


bubble. 
Actions can take at this range. 





Table 40: What did clicking in the Response Manager mean 
to you? (Question 28) 





Orders were given to do the three task clicked 
e Only that the task has been completed by someone else 
e Actions have been taken. 


e Issued track a warning, took a self-defense measure with 
CIWS, checked for territorial airspace. 


e Told CIC that were in level 1 & switch to CIWS auto. 
e It has been done. 
e Should be being carried out. 


e I am verifying that my airspace is clear, issuing a level 
1 warning to the contact and setting CIWS into auto. 


e Memory aid not a command to change CIWS status a display 
of current status. 


e Click: issue order to appropriate station. 













e No idea. 
When within range the selected actions will take place. 






Zs 


Table 41: What does the greyed-out area, change CIWS to 
auto/ready mean to you? (Question 29) 


Already ordered. 

Only that the TAO thinks it's done. 

DSS is recommending this course of action. 

Means the system is standing by to fire on a potential 
threat. 

That doctrine is being followed -active. 

That 1s a ship system, it is already in that state or 


been clicked. 

Means it is already complete w/regard to 7013. 

The CIWS will automatically track and the system is 
operational. 

Means that my CIWS status is auto/ready. 

The CIWS is still in auto/ready. 

It's been accomplished. 

Places CIWS in auto/ready. 





Table 42: What is the relationship between the Track 
Profile and the Response Manager? (Question 31) 


They are on same scale 

Range of contact, engagability vs. defensive steps taken 
by own ship 

Range of COI is the same top shows physical engagement 
limits bottom shows ROE and weapons, posture 
recommendations. 

One displays possible decisions to make on a track and 
the other provides a visible aspect of that decision. 
Top-gives threat envelopes and the track bottom gives 
what doctrine to folkew according to. themerack.. 


Show points in relative distance that require/suggest or 
move points of action. 


Drstance axis. 


Spatial in that as the contact further progresses in the 
envelope certain tasks must be completed. 

They show that a contact 1s within my weapons range and 
tracks the ROE that I have followed. 

Range of contact to ship. Range of Recommended actions. 
They are covered by range of target to own ship. 
Represents the actions that need to be taken. 

Both display range to target ship. 
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Table 43: What does the Response Manager white line mean 
to you? (Question 32) 


Actual range. 
Range to own ship. 


COI range. 

Range of contact to ship, also gives an indication of 
where you should be in your decision making process. 

The doctrine to follow according to that specific track. 


Present distance. 

Distance target is from ship. 
Distance line. 

Decreasing range of contact. 


Range of contact in question. 
Range to target ship. 





Figure 27 shows the post-task rating average for 
each of the participants. For example, the bar farthest 
to the left of Figure 27 shows that the average rating 
given across all post-task questions by this participant 
to be 3.0. Figure 28 shows the average post-task rating 
for each post-task question. For example, the bottom bar 
representing question one indicates that the average 
mat Lag i Ove this question was approximately oe 
Questions for which the rating exceeded the usability 


criterion of 3.0 include questions: 


e 13 - How easy/difficult was it to determine 
whether track 7013 was within its weapons release 
range. 

e 17 - How easy/difficult was it to identify the 
most recent warning information for track 7011? 

e 20 - How easy/difficult was it to understand what 
the numbers in the alert window mean? 

e 21 - How easy/difficult was it to understand what 


the colors in the alert window mean? 


io 


on 


- 


NO 
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Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 
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Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 27: Average Participant Ratings with Standard 
Deviation 


Questions 





a a (a a a) aa a a SS a 
0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 


Ratings 
(1-easy, 3-somewhat easy, 5-somewhat difficult, 7-difficult) 


Figure 28: Average Question Rating with Standard 
Deviation. 
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Immediately following the usability evaluation 
participants were given a post-test questionnaire. 
Figures 29 through 42 show participant ratings for ease- 
of-use and satisfaction for the DSS and its’ components. 
All ratings were within the established usability 
criteria with the exception of the Track Profile. The 
Track Profile component received an average ease-of-use 
hot OL 3.2 NanGeepartadecipant Satiskacti#enm rating of 


ore: 


qT 


Ratings 
(l-eaay, J-oomewhel eaay, 5-somewhot dificult, 7-ditticutt) 





Lower Non-Aegis Upper Nor- Aegis Lower Aegis 
Participants 


Figure 29: How easy/difficult was it for you to use the 
Geoplot? (Question 1) 


Table 44: How could the Geoplot be improved? 


Own ship speed leader. 
Switch +/- on range scales. 
Good geoplot, detailed digital maps would be excellent. 


Declutter button like the one on the JMCIS. Add symbology 
Co tHe plore 

Zoom area option. Center ship option. Speed leader on own 
ship. 

Be able to choose center instead of always own-ship. 
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Ratings 
(t-eeay, 3-aomewhat eoay, 5-eomewhat ditficutt, 7-difflcut) 
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Partiapants 


Figure 30: How satisified/dissatisfied were you with the 
Geoplot? (Question 2) 
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Ratings 
(1-eeey, 3-eomowhel easy, 5-eomewhat ditficutt, 7-ditticult) 





Lower Non-Aegis Upper Non- Aegis Lower Aegis Upper Aegis 


Figure 31: How easy/difficult was it for you to use the 
Minicros? (Question 3) 


Table 45: How could the Minicros be improved? 





Limit information on Minicro so more can be displayed. 
e Hold training on capabilities. 


e Use labels for course, speed, range, altitude, and 
associated units. 


e Did the bearing and range represent CPA data or the 
contacts bearing and range? 


e Threats need more markers, anything w/closing CPA needs 
to be marked in some manner. 


e What happens if list is too much? 
e Labeling of Range/Dist or CSE/Speed. 
Display track course and speed. 











Ratings 
(1-eeay, 3-eomowhel cesy, 5-somewhet difficult, 7-difficult) 
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Figure 32: How satisfied/dissatisfied were you with the 
Minicros? (Question 4) 
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Ratings 
(1-ea8y, J-somawhst asey, 5-eomawhat difficult, 7-ditficult} 
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Lower Non-Aegis Upper Non-Aegis Lower Aegis 
Participants 
Figure 33: How easy/difficult was it for you to use the 
Track Proflile? (Question 5) 


Dieeeahce 

Make own ships weapons a pull-down menu w/o continuous 
user control of mouse. 

Aspect and weapon buttons were a little difficult to 
understand at first. 

The “aspect” picture seems to be the better one to use. 
Suggest switching the two pictures. 

Need weapons envelopes on top down like aspect. Needs to 
be primary interface. 

Remove history, add speed leader. Use different color for 
history and speed leader. 

Unsure how to interpret. Training. 

Too many choices. All weapons on 1 display. 





Ratings 
{1-casy, J-somowhet easy, 5-aomawhat difficut, 7-ditficult) 
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Figure 34: How satisified/dissatisfied were you with the 
Track Profile? (Question 6) 
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Ratings 
{1-oasy, 3-aomewhat easy, 5-aomewhat difficult, 7-ditflcutt) 





Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 35: How easy/difficult was it for you to use the 
Toolbar? (Question 7) 


Table 46: How could the Toolbar be improved? 


Complete NTDS symbology. 

Needed training. 

Make threat color red. 

+ and ~ range buttons. 

Change clicks into distinct visible color changes. 
Add a select all, remove the sound when selecting. 
Be able to combine background elements. 

Make it so I can choose my own magnification. 








Ratings 
{t-oasy, 3-aomowhot eosy, 5-aomewhet diftlcul, 7-ditflcuk) 





Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 
Participants 


Figure 36: How satisified/dissatisfied were you with the 
Toolbar? (Question 8) 
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Ratings 
(1-eaey, 3-somswhat sasy, 5-somawhat difficult, 7-difficult) 
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Figure 37: How easy/difficult was it for you to use the 
Response Manager? (Question 9) 


Tellme @ele these are adirecere ves or Leoconmenaatrons. 


Excellent doctrine tool. 


I think feedback on time action complete could be added 
to each line, so TAO knows it’s done. 


If distance line passes into the area, have the area 
color stand out until task is completed. 


Unsure what it is used for. Training. 





Ratings 
(1-eaey, 3-somawhat assy, 5-somawhat difficun, 7-ditflcut) 
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Figure 38: How satisified/dissatisfied were you with the 
Response Manager? (Question 10) 
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Participants 


Figure 39: How easy/difficult was it for you to use the 
Track Summary? (Question 11) 


Training. 

Allow user to manually type in track #. Need to add time 
to CPA and/or time to COI at weapons release range. 

CPA was a Tattle datficult "80 atch at fatest. 

T had trouble finding the CPA. 

Track selection is hard to discover needs marker, so you 
can tell it’s a button. 

Add for keyboard numerical pad change of contact. Further 
explain CPA by adding CPA bearing and CPA range lines. 
Clicking and Hholdinag von the track # toudisplay 
info/select a new track did not seem natural to me. 
Increase size of own ship CRS and SPD. Add own ship speed 
leader. 
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Ratings 
{l-easy, 3-somewhat easy, S-eomewhet difficult, 7-ditticutt) 
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Figure 40: How satisfied/dissatisfied were you with the 
Track Summary? (Question 12) 
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Ratings 
(1-eeey, 3-eomewhet eesy, 5-eomewhet difficult, 7-diffleuh} 





y 
Lower Non-Aegis Upper Non- Aegis Lower Aegis 
Participants 





Figure 41: Overall, how easy/difficult was it for you to 
use the DSS? (Question 13) 





Ts ideale Ghee 

e Pull-down menus for alerts and own ships weapons. 

e Training, more use of color or shape as indicators, 
rather than sound. 

e Very high learning curve. Vast improvement to tactical 

display and management. 

Ensure track number does not block symbology. 
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Figure 42: Overall, how satisfied/dissatisfied were you 
with the DSS? (Question 14) 
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Table 47: What changes would you like to make to the 
DSS? (Question 15) 


Faster response to “weapon” (upper right) and have an 
alert to the right of minicro that says “more contacts”. 
Can you modify the Response Manager for changing ROE, 
optasks, or missions such as amphibious assault or 
maritime interdiction ops? 


+ and -— on range scales reversed. 


Colors are “eye-catchers” and even have meanings. Use 
very descriptive words. Change track profile picture to 
the “aspect” picture that was clearer to me. Place track 
data in table format (this may save space and be easier 


to look at). 

Change weapon envelopes into geographic aspect display to 
account for weapon’s masking. 

Allow keypad input for the contact number in Track 
Summary. Add titles to CPA BRNG and CPA RNGE and add an 
estimated time to CPA. Add a plus or minus to the alert 
times, not how long after the alert was posted. 

Slight interface change. For example, have the menu stay 
up after you click the alerts for a contact. 





Table 48: What were the worst aspects of the DSS 
interface and why? (Question 16) 





Split screen. 
No time to CPA. No time to COI w/in its weapons release 
range or w/in your weapons release range. 

Increase the size of the “CPA” indicator. Weapon pull- 
down menu is difficult to use. 

Track profile picture is still a little confusing. Keep 
things simple. 

No idea about some features because they give no 
indications they are buttons. 

Too much reliance on the mouse. With 2 screens and 
looking in other areas besides the 2 monitors, the mouse 
arrow is easily lost. 

Weapons button was difficult to find and understand. 
aspect display. 





History in weapons display, 





Cis 


Table 49: What were the best aspects of the interface 
and why? (Question 17) 








Fase-of-use. 
Good data, lots of info displayed here. 

Easy to see threats and weapons envelopes. 

The decision making process time line with target range. 
Use of colors. All on “1” screen. It was all right there. 
Geographic w/map display is nice. 

Minicro-I like the quick and pertinent information on the 
contact. Response Manager-excellent doctrine aide. 
Aspect-allows for quick maneuvering recommendations to 
OOD. 

Overall I think it was an easy to use interface. 

Liked having so much information simultaneously visible. 











Table 50: Where there any parts of the interface that 
you found confusing or difficult to understand? 
(Question 18) 


Multiple threat displays possibly pop-up, sizeable 
windows. 


No, not really. 

Not really, Gitireul tives orinaamiamcue co Unitama do aa Tey 
w/system. 

It took me a few minutes to understand aspect. 


Track profile picture - it showed weapon engagement 
envelopes but I’m still a little confused by it. The 


“aspect” picture was clearer to me. I’m not sure what the 
track profile picture axes were? 


Time late on the alert box needs to be 00:00:00 format. 


Adding new contact as far as reporting one that 1s not on 
the minicro. It is easy to get used to the display and 
not look at the geoplot. 


Weapons button. Track summary. 





Track profile area. 
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V. SUMMARY, CONCLUSIONS, AND RECOMMENDATIONS 


A. SUMMARY 


The DSS did not meet these usability objectives of 
90% across all tasks for task completion. The overall 
task completion rate across all tasks was 84%, 6% below 
the usability criterion initially established. When 
examined by individual task, on 13 tasks the DSS 
Surpassed the 90% usability criterion level and on 10 
tasks the DSS-2 did not meet the usability neers 
The only task on which all participants committed an 
error was the identification of the map’s upper and 
lower range scale. Half of the participants were unable 
to properly identify potential threats and one-third of 
the participants had difficulty displaying all the track 
numbers on the map display. A quarter ote all 
PAatrmerlpDanesmCOMMMmetea Crrors “on Viave different “tasks. 
These tasks were: 


e Display all surface unknown tracks; 

e Change the size of the map to better see the 
tracks displayed; 

e Check what the most recent warning information is 
regarding track 7011; 

e Determine whether track 7013 1s within its 
weapons release range; 

e Determine whether track 7016 was within its 
weapons release range using the track profile 
component. 
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Two participants incorrectly identified the most recent 
WarhangG@intormation for trae Oli 

Overall, the DSS-2 did not meet the 90% usability 
objective across all tasks for ease-of-use. Eighty-two 
percent (8% below the 90% objective) of the ease-of-use 
questions averaged a rating of somewhat easy or better. 
The four questions that did not meet the ease-of-use 
usability objective goals were: 


© How easy/difficult was it to determine whether 
track 7013 was within its weapons release range? 

e How easy/difficult was it to identify the most 
recent warning information for track 7011? 

e How easy/difficult was it to understand what the 
numbers in the alert window mean? 


e How easy/difficult was it to understand what the 
colors im the alert window mean ? 


Four ease-of-use questions received a rating of six or 
higher by at least on participant, where a rating of 
five corresponded to somewhat difficult and a rating of 


seven corresponded to difficult. These questions were: 


e How easy/difficult was it to display all tracks? 

e How easy/difficult was it to identify whether track 
7013 was within its weapons release range? 

e How easy/difficult was it to understand what the 
numbers in the alert window mean? 

e How easy/difficult was it to determine whether 
track 7016 was within it's weapons release range 
using the intormation, displayed in the right 
TemL tol ? 
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iigeomusability objective for satisfactmwon was met,«100% 
of satisfaction ratings were below a rating somewhat 
satisfied. 

During the usability = study, participants were 
questioned about the functionality and user interface 
design of some components of the DSS-2. The following 
information was compiled from participant’ responses. 

The range scale feature presented some difficulty 
to participants. Participants were able to correctly 
pear how the range magnification buttons functioned 
after exploring the button. However, seven participants 
stated that they believed that the labeling of the 
buttons was reversed. 

When questioned about the ordering of the minicros, 
all but one participant believed the minicros were 
ordered in some manner. Of the eleven participants who 
believed the minicros were ordered, nine correctly 
assumed that they were ordered by threat. When asked to 
select a track that did not have a corresponding 
Nan2ere,  fOur Ors tEnem=Il2 participants committed an error. 
A majority of participants expressed surprise that some 
tracks did not have a corresponding minicro. 

POI eee Dparerectpants had difficulty selecting the 


Glee Em@mDOUtcCOmm—mon the mingecro. In addition, although 
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eleven participants correctly identified the numbers 
found in the alert window, every participant expressed 
frustration that these numbers were not labeled. Four 
participants stated that additional or different formats 
for these numbers might be more appropriate. In general, 
participants expressed significant dislike for the way 
the alert buttons had to be selected. The interaction of 
Having to click “and Heolaw'the button &o wead the 
information contained within the pop-up window caused 
f£rustratwen. Many participants ie ale te that His 
interaction would be inappropriate during shipboard 
operations. 

The aspect loyblieco}iel surprised almost all 
participants. Ten participants stated that this button 
did not meet their expectations, however, nearly all 
participants liked the feature. The predominate 
criticism of the aspect button was its size, many 
participants expressed that it was too small. 

The post-test questionnaire examined the 
participants overall experience with the DSS-2 and with 
each individual component. Overall, results of the post- 
test questionnaire suggest that the DSS-2 met the 
usability objective criteria imeietally established of 


somewhat easy or better Lon ease-of-use and 
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Satisfaction. The average DSS-2 ease-of-use rating was 
2.6 and the average satisfaction rating was 2.3. The 
only DSS-2 component that did not meet the ease-of-use 
and satisfaction usability objectives in the post-test 
questionnaire was the track profile component. This 
component received a Se fOr ease-of-use and 
Samusfactwon., All participants reported that they 
enjoyed working with the DSS-2 and believed that it 
would be a Significant step forward in @PC information 


Management. 


B. RECOMMENDATIONS 


Table 51 details the usability issues encountered 
bao Gudy, spalLelecipames Cuming this usabalaty valuation 


and provides corresponding recommendations. 


Table 51: DSS Component Issues and Recommendations 


RECOMMENDATION 


Geoplot Track symbology 
1. Some participants did not like meconauct further 
Ehemeoloring of potential research to 
threats and unknown tracks. determine the 
Participants stated that optimal use of 
unknowns were potential mracktweo lor. 
threats. 


Toolbar Track Numbers 
1. Some participants wanted the . Change existing 
track number button’s name to button label. 
be changed to “Display all 
tracks #’s”. 










Range Magnification 

1. All participants had 
difficulty discovering jene 
upper and lower bounds of the 

range scale. 











1. Remove button 
option and 
replace with a 

slider widget. 

















Change the range 
magnification 
icons relative 
posttpzens.. 


Many participants chose the 
incorrect range magnification 
button to resize the geoplot. 





















Provide numerical 
keypad capability 
to enter a 
Specific range 
Magnification. 


Some participants wanted to be 
able to enter any range using 
a numerical keypad. 
























Provide this 
dynamic map range 
selection 

capability. 


One participant wanted to be 
able to use the mouse on the 
geoplot to select an area to 
be magnified by clicking and 
dragging across the area. 
General 
1. Many participants wanted 
abbreviated labels on the 
information displayed. 










Minicro 





1. Provide more 
labeling within 
the alert window. 


















Maintain the 
selected state of 
the minicro, even 
if va pr wer wey 
changes. 
POTENTIAL 
DANGEROUS. 


Several participants were 
Surprised when the selected 
minicro changed. A few users 
looked at another area of the 
DSS-2 and did not expect the 
selection to change when the 
Dior wey cna: 




























Alert Information 
1. Many participants expressed 
frustration when they could 
mot bring up the alewe 
quickly. Participants clicked 
on the alert button and didn’t 
realize they had to click and 
Hold 1t to brenguwe alert 
window. 











1. Redesign the 
alert selection 
mechanism so 
that one click 
will open alert 
window anda 
second click 
wed | Close it. 























Label 
information. 


Many participants wanted the 
Information. Contameaea in tene 
alert window to be labeled. 



















Test the 
feasibility of 
having both 

Opt LOns. 


Some participants wanted the 
time of the alert, as well as 
the time since the alert, to 
be displayed. 















Additional 
research on 
Go Our 


Many participants did not like 
the light blue color and 
preferred for red or yellow. 
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Weapons Envelope 

1. Some participants had 
Cekpruculty identifying when 
weapons envelope, red or blue, 

related to the selected track. 















1. Provide training 
and easily 
accessible help 

feature. 


Profile 

















Provide the 
capability to 
have multiple 
weapons 

envelopes. 


One participant wanted the 
ability to have multiple 
weapons envelopes displayed 
Simultaneously. 






One participant had diftticuley Rite Wc 
identifying the location of 


own-ship. 
















Aspect Insert 
1. Many participants stated that 
the aspect button should be 

labeled “Weapons Cut-Out” or 
*CUESOUE”. 







1. Redesign and test 
alternative 
labeling. 













Increase size of 
Aspect Insert. 


Many participants wanted the 
Aspect Insert to be larger. 










Some participants preferred Tr asemenC . 
the Aspect Insert to the Track 


Prefile. 












General 

1. Many participants stated that 
they were unsure whether the 
options in the Response 
Manager were recommendations 

or requirements. 










Response 
Manager 







Soe ye ciel t,t 5) Chae 















Existing feature 
that was not 
tested. 


Many participants wanted to be 
able to edit the Response 
Manager. 





















Track General 













Summary 1. Some participants had 1. Redesign and test 
difficulty reading the alternative 
information contained in the Colors: to 
Track Summary due to poor facilitate 





background and text coloring. reading. 


CG. CONCLUSIONS 
This study examined the usability of the DSS-2 ina 
systematic manner and established where the DSS-2 


janqreelblele is today against usability criteria. 
oS 


Furthermore, participants identified usability issues 
that can be addressed in future design and research 
efforts. Based on the information obtained from this 
evaluation, recommendations to address the usability 
issues were made. Overall, in terms of ease-of-use and 
Satisfaction, participants reacted positively to the 
DSS-2. Participants felt the DSS-2 would assist them in 
Maintaining Situational awareness and was a tool that 
would be useful onboard Navy ships. 

ae methodology applied in this study was useful in 
the evaluation of the DSS-2. This study demonstrated 
that traditional human-computer interface usability 
methods could be directly applied the evaluation of 
synthetic environments. The DSS-2 is a simple synthetic 
environment represented on two computer monitors. Given 
the success of this methodology with the DSS-2, it would 
be appropriate to use this methodology in evaluating 


more complex synthetic environments. 
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APPENDIX A: CONSENT FORM 
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CONSENT FORM 


Usability Evaluation of the Decision Support System 


Principal Investigator: LT Dylan Schmorrow 
Operations Research Department 
Naval Postgraduate School 
Monterey, CA 93943 


I, , consent to my participation in the research project titled 
Usability Evaluation of the Decision Support System. 


| understand that | am free to withdraw my participation in the research at any time and that if | do 
| will not be subjected to any penalty or discriminatory treatment. 


| have been given the opportunity to ask questions about the research and received satisfactory 
answers. 


| understand that any information or personal details gathered in the course of this research about 
me are confidential and that neither my name nor any other identifying information will be used or 
published without my written permission. 


| understand that if | have any complaints or concerns about this research | can contact: 
George Conner 
Operations Research Department 
408-656-3306 


Signed by: 


SHRSSHSHSSHSHSSHEHSSHSHSHSHSHSHHSSHSHSHSHSHSSHSHSHHSSSSHSHSSHSHSSHSHSSHSHSSHSSSSSSEHH ES SESSESB SORES 


@eeeeevmeneeseeeeeeeseeeeeeeeeeeeeee ee eeeeeseeeeeeeeeeseseeeeee ee eeeeeees 


S¢ 


APPENDIX B: DEMOGRAPHIC INFORMATION FORM 


ov 


DEMOGRAPHIC INFORMATION 


Name: Rank: Designator: 
Years of service: Time in rank (months): Curriculum: 
Previous Command: Primary billet: 
Months onboard: Months in shipyard (if applicable): 
Months standing CIC watch: Months standing TAO watch: 
Hours spent on computer each week: Operating system primarily used: 
Qualifications: Circle One Date 

CIC Watch Officer? Yes No 

SWOS Department Head Course? Yes No 

SWC Qualified? Yes No 

STWO Qualified? Yes No 

TAO Qualified? Yes No 

Aegis Qualified? Yes No 

Other? 


Experience Summary -- List assignments relevant to SWO experience 
(include Combat Systems) 


Command Months 


Deployments Frequency 


LANT 
PAC 

MED 
PERGULF 
Others 
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APPENDIX C: TASK SCRIPT 
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Background 


This project was spawned by the 1988 USS Vincennes 
incident where an Aegis cruiser engaged in a littoral 
warfare peace-keeping mission shot down an Iranian 
Airbus. Investigations following the incident suggested 
that stress may have affects on decision making, and 
that these effects were not well understood. ids 
project was established to address these concerns. 

This prototype Decision Support System (DSS) was 
developed to enhance Navy tactical decision making based 
on “naturalistic” decision processes. Displays were 
developed to support critical decision making tasks by 
Naval watch officers operating in a shipboard Combat 
Information Center. 


SCENARIO 


This scenario has the ship operating independently 
in the northern Persian Gulf 50 nm to the east of Kuwait 
Ciscy = You are on a presence patrol and have been 
Gdizrected to remain WilthunsS Nm Of =your CcUmmence posit 1loOm 
to demonstrate US resolve. Weapons and warning status: 


Yellow and Tight. At scenario start you are on course 


O70, speed 7 knots. Wal Sasa eter es reduced to 
approximately 4 to 5 nm in dust and haze. Local time is 
OO 
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Task 1 


Display the track numbers of all contacts in the map 
display. 


Task 2 


Locate and select track number 7012. 


OA: 


Post Task Questions 


1. How easy/difficult was it to display all the track 


numbers? 
J a | 
1 Zz 3 4 5 6 7 
easy somewhat somewhat Qi FETE weet 
easy garirecult 
Comments: 


What might have made this task easier? 


2. How easy/difficult was it to find track number 7012? 


i Z, 3 4 5 6 7 
easy somewhat somewhat Garr Pace were 
easy Glerreutt 
Comments: 


What might have made this task easier? 


oem Hew casyvraiiimeult was it to read the track number on 
the map display? 


i Zz | 4 5 6 7 
easy somewhat somewhat ebm voult 
easy ner nem 1 t 
Comments: 


What might have made this task easier? 


Task 3 
Change the map to display the directions all tracks are moving. 


Task 4 


Remove all unknown tracks from the map display. 


Task 5 


Display all surface unknown tracks. 


Task 6 


Display all tracks. 
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Post Task Questions 


4. How easy/difficult was it to display the course leaders 
in the map display? 


a Z 3 4 5 6 a 
easy somewhat somewhat Gabe feu it 
easy Gate Sncuie 
Comments: 


What might have made this task easier? 


5. How easy/difficult was it to remove unknown tracks from 
the map display? 


alt 2 5 4 5 6 7 
easy somewhat somewhat Galt t 1 Cube 
easy hie meecw) t 
Comments: 


What might have made this task easier? 


LoS 


6. How easy/difficult was it to display all surface unknown 
tracks on the map display? 


i Z 3 4 5 6 7 
easy somewhat somewhat Gabe Ea cule 
easy Grete te 1G uele 
Comments: 


What might have made this task easier? 


7. How easy/difficult was it to understand when the track 
type buttons (1.e., surface, unknowns) were selected. 


il 2 S 4 5 6 7 
easy somewhat somewhat gait t veut. 
easy Gier meult 
Comments: 


What might have made this task easier? 
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Task 7 


Change the size of the map to better see the tracks 
displayed. 


Task 8 


Identify the range of map sizes that are available. 
Read aloud the upper and lower range levels. 


Task 9 


Explore these two buttons. When finished exploring, select 
map size of 128 nm. 


Belt eet tse 





Oy 


Post Task Questions 





8. How would you explain what this button does 





9. How would you explain what this button does 


10. Does the positioning of these buttons (plus on the 
left / minus on the right, meet your expectations? 


Yes / No 


Leno Sawoy noe 
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Task 10 


Please point to own-ship on the map display. 


Task 11 


Tdentify which tracks are potential threats? 


Task 12 


Determine whether track 7013 is within its weapons release 
range. 


Task 13 


Determine whether track 7013 is within own-ship’s weapons 
envelope. 


ie 


Post Task Questions 


11. How easy/difficult was it to identify own-ship on the 
map display. 


it 2 3 4 5 6 y 
easy somewhat somewhat Girr Lewes 
easy Gaset 1 Cit 
Comments: 


What might have made this task easier? 


12. How easy/difficult was it to identify which tracks 
were potential threats? 


i 2 8 4 5 6 7 
easy somewhat somewhat Garr reulie 
easy Gisetere1y | t 
Comments: 


What might have made this task easier? 


a0 


13. How easy/difficult was it to determine whether track 
7013 was within its weapons release range? 


il 2 3 4 > 6 7 
easy somewhat somewhat Giger FCU1t 
easy Govmmien1 
Comments: 


What might have made this task easier? 


14. How easy/difficult was it to determine whether track 
7013 was within own-ship’s weapons release range? 


q 2 2 4 5 6 7 
easy somewhat somewhat Seem 1 Ccult 
easy aa firewle 
Comments: 


What might have made this task easier? 


dpe 


Task 14 


Identify and read aloud the bearing and range of track 
70 ioe 


feiZ 


Post Task Questions 


15. How easy/difficult was it to identify the bearing and 
range of track 70167 


a 2 iS 4 5 6 7 
easy somewhat somewhat Ghee eu tc 
easy @a. i Bake let 
Comments: 


What might have made this task easier? 


Bee 7037 fa 7001 0 
ek ak Unknown 
Corgbattante : Lal eee 


079°/15.2 Bice 190°/8.9 HH o71*/21 O41°/34 


Surface Masui m Surface 725000 §% + 3000 
gs | : Re ES 





16. Do you think there is any meaning to the ordering of 
these items? 


YES / Ne 


If yes, what do you think the ordering means? 


ors 


Task 15 
The DSS is continually updating information on the behavior 


of all tracks. Check and see what the most recent warning 
information 1S regarding track 7011. 


Task 16 


Identify and read aloud the most recent warning information 
POC eee ee Cele 
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Post Task Questions 


17. How easy/difficult was it to identify the most recent 
WarnInGLimnonnation Lor track 70d? 


en eee | | |__| 


1 2 3 4 5 6 7 
easy somewhat somewhat Geer iculte 
easy Gabe t 1eulLt 
Comments: 


What might have made this task easier? 


18. How easy/difficult was it for you to select the alert 
button and view the alert window? 


alt 2 3 4 5 6 7 
easy somewhat somewhat Gath eve Lt 
easy alae te fetes 
Comments: 


What might have made this task easier? 


19. Click and hold one of the alert buttons. 
A. What do the numbers on the right mean to you? 
Time elapsed since warning occurred. 
Time warning occurred. 
Other, please described 


B. What do the colors mean to you? 


iS 


20. How easy/difficult was it to understand what the 
numbers in the alert window mean? 


i 2 B 4 5 6 7 
easy somewhat somewhat emEticult 
easy Gmetticult 
Comments: 


What might have made this task easier? 


21. How easy/difficult was it to understand what the 
colors in the alert window mean? 


a Z 3 4 5 6 7 
easy somewhat somewhat di feweult 
easy Git ter 1 C 
Comments: 


What might have made this task easier? 
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Task 17 


Select track 7016. 


Task 18 


Locate own-ship symbology on the right monitor. 


TASK 19 


Using information available on the right monitor, determine 
whether track 7016 is within its weapons release range. 


TASK 20 
Using information available on the right monitor, determine 


whether track 7016 is within own-ship’s 5/54 Guns weapons 
envelope. 


oe 


Post Task Questions 


22. How easy/difficult was it to determine whether track 
7016 was within its weapons release range uSing the 
information displayed in the right monitor? 


1 Z 3 4 5 6 7 
easy somewhat somewhat da fiveul & 
easy ieee cult 
Comments: 


What might have made this task easier? 


23. How easy/difficult was it to determine whether track 
7016 was within own-ship’s weapons release range using 
the information displayed in the right monitor? 


i 2 8 4 5 6 7 
easy somewhat somewhat Cosmet LCudite 
easy di Pewee 
Comments: 


What might have made this task easier? 


ames 


24. What@deomyoume-peeceathne ASPECT button to do? 


25. Click it. Does this meet your expectations? 
Yes/no 


iene, Why Mor 2 


26. How satisfied/dissatisfied are you with the ASPECT 
button feature? 


i 2 3 4 5 6 7 
satisfied somewhat somewhat dissatisfied 
satisfied dissatisfied 
Comments : 


ier) 


Post Task Questions 
Task 21 


Select track 7017. 


oe ae we. Poe | eee AS nt 
ee = - es _— anda a 
Verify oirspoce 

PES eT eit: 


CiwS to auto/readg Sell idefense systems 


2nd -worning “ 


is 
- Execute EW pach ages 


es Report to senior 
_ Alert Stinger Datachmen! 
Depioy'Decoss | 


Fire Fleres/werntng Sho! 
__ Engoge / De ast Engagn iecision ~~ 


Question 27: What would you expect to do with this area 
based on the track selected? 





Task 22 


Question 28: Click on verify airspace, issue a level 1, and 
change CIWS to auto/ready. What does it mean to you? 


Task 23 
Select track 7013 


Question 29: Please describe what the dark gray area, “CIWS 
to auto/ready”, means to you. 


Did you expect the “CIWS to auto ready” to be dark gray? 
WES 7 ae 


Are the actions listed here REQUIRED or RECOMMENDED? 


a0 


Ss TRenpeerew POS acl), 


a> cole a 


ee 


EEE IED 


so sdiemdapaahiiid thane Se ees eee 
ace BOD aeRO | Fae at cae cree Beh a aoe 


a a | rie ae che % Le ae 
Me ADT PIG OO EN IT od Fone OT S Sa ; : 


Me meer oes Sc 


er 


a 


t 

x - 
7 

4 

A 
a 

I 

7 

I 

4 


ited PC LUys ready Reali ieee Tilia oe 


nt at ee eee eae ei 


Be Cie, warning. Se Oe) 


fai Cover 
; é ey Execute EW packages 
| and warning 
HWluminate 
Report to senion 7) oe _ Reper ae Senior 
Alert Stinger Detaci iment 
Deploy Decoys 
Fire Flares/Warnint Shot 
Engage / Do not En age decision 





Question 31: What is the relationship between the two areas 
above? 


Question 32: What does the white line represent to you? 


it 
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APPENDIX D: POST-TEST QUESTIONS 
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OVERALL POST-TEST QUESTIONS 





= a 


Figure 43: Geo-Plot with Desaturated Map and Variable 
Coded Symbology 


1. How easy/difficult was it for you to use the Geo-Plot? 
a eae | | ee ee eee | 
a: z 2 4 5 6 y 
easy somewhat somewhat Ge fet 
easy Gerace 
Comments: 


How could this area be improved? 


2. How satisfied/dissatisfied were you with the Geo-Plot? 
se ase ey | neocon | ce ee |e [eee | | 
a: ep 3 4 5 6 7 
satisfied somewhat somewhat dissatisfied 
satisfied arssaers tires 
Comments: 
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7013 0 Me 7037(NN 867001 © 70235 § 7020 - 
loss Bee) =6lnknown P-$ Be Helo/Lt Air 
Combattante! . Caos | 


079°/15.2 161°/27 190°/78.9 | O71°/21 ' 041°/34 


Surface Breit Surface ~ 5000 ea 7 5000 
Caster Hl ere 8 Decca-1226 APS-115 ff Me ES 
2s io FF Mo IfF No IFF 


ask (alerts) alert | 





Figure 44: Sample Mini-CRO 


3. How easy/difficult was it for you to use the Minicros? 


1 e 5 4 5 6 7 
easy somewhat somewhat @iiiefets we dt 
easy eee fivcult 
Comments: 


How could this area be improved? 


4. How satisfied/dissatisfied were you with the Minicros? 
1 2 3 4 5 6 7 
satisfied somewhat somewhat dissatisfied 
Sabackied dissatisfied 
Comments: 


LZ 


cele RCL Fy) 


| Stab Las 


Pee ers 





\ ] i | t 
Se ee eS ee 


Figure 45: Track Profile with Aspect Inset 


5. How easy/difficult was it for you to use the Track 
Profile? 
__ a ee ee | | 
a Z 3 4 > 6 ch 
easy somewhat somewhat Gitsmereu lt 
easy Gant eee |e 
Comments: 


How could this area be improved? 


6. How satisfied/dissatisfied were you with the Track 
PEOLIT Le? 
[a ee | 
4 Z 3 4 5 6 a 
SatwsEtred somewhat somewhat dissatisfied 
satisfied aGrssabrustied 
Comments: 
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4 o terete 
RM 4: : se eet Zs 
ae See we 
oem 8 


Tee 


od 


a ait aoe 
48: och, 


“3 | 
3} red a 
= 


Track Types 
Pie eee 
bees ic} rm tJ 
inteeves] TS] 


- Range Scale 


Cure ee 





7. How easy/difficult was it for you to use this area? 


ale Ze 3 4 5 6 7 
easy somewhat somewhat difficult 
easy Cater are us 


Comments: 


How could this area be improved? 


8. How satisfied/dissatisfied were you with this area? 


1 2 3 4 5 6 oy 
satisfied somewhat somewhat dissatisfied 
satisfied dissatisfied 


Comments: 


see 
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eete eis 
EST ey 


Fire Fleres/warning She! 
Engage / Dc not Engage tiecision 


Figure 46: Response Manager 





9. How easy/difficult was it for you to use the Response 
Manager? 
|____|____ |e | || 
i ae 3 4 5 6 7 
easy somewhat somewhat Gaetano alee 
easy Git pen Lt 
Comments: 


How could this area be improved? 


10. How satisfied/dissatisfied were you with the Response 


Manager? 
ik 2 3 4 5 6 7 
satisfied somewhat somewhat dissatisfied 
satisfied dissatisfied 
Comments: 
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a ee oS ee ee neaskotes 


LW ‘ex: vA ripe ea at Take aad ‘Asses 


i denemeen soa 
‘Bedring| ya a7 


: MENG Caan 


TE) 


400 ~- 3000 : 
Primus-40 


a ae we 


aE J tay as ta 





11. How easy/difficult was it for you to use the Track 


Summary ? 

ee ee | | eae | 

ul 2 3 4 5 6 7 
easy somewhat somewhat Qmbircult 


easy @eitficult 
Comments: 


How could this area be improved? 


12. How satisfied/dissatisfied were you with the Track 


Summary ? 
ey en (oe. AA ee | | 
il 2 3 4 5 6 7 
satisfied somewhat somewhat dissatisfied 


satisfied prssatcisried 


Comments: 


Zo 


im eed 


ck Ce ee eee CL ed 
et Me | leh ei es Lae 
Fa ea! ) a a 


ee 


te 
e 


owe Vtares/worning 1 bet i 
Cagege / Be ~«' Cage pe Gurincen 


ah Lid 





Figure 48: 


ons as ars sue 


TADMUS DSS Integrated Display 





13. Owerall, how easy/diffacult was it dor you to use the 


DSS? 
Ee a 
1 2 3 4 5 6 7 
easy somewhat somewhat Gm LIT CULT 
easy Gurl cult 
Comments: 


How could this area be improved? 


14. Overall, how satisfied/dissatisfied were you with the 


DSS? 
ee ee a 
il Zz 3 4 5 6 7 
satisfied somewhat somewhat dissatisfied 
satisfied GuLSSyelie i enmaleve| 
Comments: 
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Question 15: If you could improve the DSS system, what 
changes would you like to make? 


Question 16: Overall what were the worst aspects of the DSS 
interface and why? 


Question 17: Overall what were the best aspects of the interface 
and why? Please describe. 


Question 18: Where there any parts of the interface that 
you found confusing or difficult to understand. Please 
Describe. Please describe. 
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